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FOREWORD 


A  significant  problem  facing  Army  trainers  today  is  the  need  to  rapidly 
establish  high  levels  of  technological  expertise  in  unskilled  trainees.  The 
development  of  programs  to  train  soldiers  with  widely  varying  aptitude  levels 
in  these  areas  in  a  way  that  they  can  quickly  comprehend  and  assimilate  will 
depend  upon  the  determination  of  how  learning  in  these  areas  occurs.  In  re¬ 
sponse  to  this  problem,  research  to  investigate  complex  learning  processes  in 
computer  technology  was  conducted  by  the  Technologies  for  Skill  Acquisition 
and  Retention  Technical  Area  (TSARTA) .  This  task,  titled  "Use  of  ICAI  in 
Training  Systems,"  was  organized  under  the  "Train  the  Force"  program  area  and 
was  supported  by  the  U.S.  Army  Engineer  School  (USAES).  These  research  re¬ 
sults  were  briefed  to  the  Deputy  Assistant  Commandant,  USAES  in  June  1*586  and 
components  were  incorporated  into  the  computer  literacy  training  program  for 
the  Engineer  Officers  Advanced  Course  in  October  1986. 


^ 

EDGAR  M.  JOHNSON 
Technical  Director 
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CHANGES  IN  KNOWLEDGE  REPRESENTATION  WITH  INCREASING  EXPERTISE 


EXECUTIVE  SUMMARY 


Requirement ; 

To  investigate  how  trainee  knowledge  representations  change  as  expertise 
in  command  language  (Lotus  1-2-3)  use  is  acquired  and  to  determine  whether 
these  representations  are  related  to  ability  to  use  the  language. 


Procedure : 

Twenty-four  MBA  students  at  the  University  of  Michigan  who  had  no  previ¬ 
ous  experience  with  Lotus  1-2-3  received  a  1-1/2-hour  tutorial  on  basic  use  of 
Lotus  and  were  subsequently  required  to  use  the  software  in  a  class  project. 
One  week  after  the  tutorial  and  at  the  end  of  the  semester  following  comple¬ 
tion  of  the  class  project,  each  trainee  participated  in  a  2-hour  data  collec¬ 
tion  session  that  assessed  their  organization  of  Lotus  knowledge  and  their 
ability  to  use  the  software. 


Findings ; 

Comparisons  of  trainee  knowledge  representations  obtained  at  the  two 
points  in  the  training  period  indicated  how  knowledge  changed  as  trainees 
became  more  skilled  in  using  Lotus  1-2-3;  comparisons  of  trainee  representa¬ 
tions  with  the  representation  for  an  expert  Lotus  user  indicated  how  novice 
trainee  knowledge  differed  from  expert  knowledge.  Over  the  training  period 
trainee  knowledge  representations  began  to  resemble  expert  representation; 
trainees  with  higher  similarity  to  the  expert  demonstrated  a  better  ability 
to  use  the  software  than  those  with  lower  similarity  to  the  expert. 


Utilization  of  Findings: 

This  research  demonstrates  that  novice  knowledge  representations  for 
computer  command  language  skills  change  as  these  skills  are  acquired  and  also 
shows  how  novice  representations  differ  from  expert  representations.  This 
information  has  been  used  to  develop  a  handbook  and  tutorial  for  Lotus  1-2-3 
that  will  be  placed  in  the  U.S.  Army  Engineer  School  computer  laboratory  (The 
Army  Engineer's  Lotus  1-2-3  Tutorial  and  Handbook).  This  handbook  and  tuto¬ 
rial  is  unique  because  it  contains  instructional  material  designed  to  address 
student  misconceptions  and  errors  that  were  revealed  by  the  present  research. 
This  research  also  shows  that  a  method  for  mapping  trainee  and  expert  cogni¬ 
tive  representations,  the  ordered  tree  technique,  may  be  a  candidate  for  in¬ 
corporation  into  an  ICAI  system  to  aid  in  student  modeling. 
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CHANGES  IN  KNOWLEDGE  REPRESENTATION  WITH  INCREASING  EXPERTISE 

INTRODUCTION 

Assessing  the  cognitive  state  of  the  student  is  a  fundamental  aspect 
of  all  educational  systems.  Historically,  however,  this  task  has  not  been 
especially  cognitive  and  has  rested  primarily  on  performance  measures  such 
as  test  scores,  number  of  problems  solved,  speed  of  performance,  etc. 
According  to  Ohlsson  (1986),  this  performance-based  approach  assumes  that 
knowledge  of  how  much  a  student  knows  is  more  important  than  what  that 
student  knows.  However,  while  this  approach  is  distinctly  non-cogni tive 
in  theory,  in  practice,  creative  teachers  combine  the  information  gained 
from  these  qualitative  performance  measures  with  an  understanding  of  the 
subject  matter  domain  and  student  learning  characteristics  to  infer  the 
cognitive  state  of  their  students. 

In  recognition  of  this  fact,  researchers  in  intelligent  computer-aided 
instruction  (ICAI)  and  in  cognitive  and  educational  psychology  have  begun 
to  explore  sophisticated  methods  that  allow  inferences  to  be  made  about 
cognitive  structures  and  processes.  Within  the  area  of  intelligent  com¬ 
puter-aided  instruction,  methods  have  been  developed  that  produce  what  is 
known  as  the  "student  model."  These  include  the  overlay  method  (Carr  & 
Goldstein,  1977),  the  mal-rule  and  perturbation  method  (Brown  &  Burton, 
1978),  and  various  simulation  methods  (Ohlsson,  1986).  In  the  overlay 
method,  performance  measures  are  used  to  determine  mastery  of  specific 
knowledge  elements  contained  in  an  internal  representation  of  a  subject 
matter  domain.  These  elements  are  marked  as  proficiency  increases  so  that 
student  knowledge  is  modeled  as  an  "overlay"  on  an  expert  representation 
of  the  domain.  In  the  mal-rule  method,  the  pattern  of  the  correct  and 
incorrect  answers  is  matched  against  a  library  of  errors  or  "bugs"  and  is 
used  to  infer  student  misconceptions  that  lead  to  deviations  from  pro¬ 
scribed  procedures.  In  simulation  methods,  the  cognitive  steps  taken  to 
reach  a  particular  problem  solution  are  modeled  and  are  then  executed  to 
determine  whether  they  reproduce  a  student's  solution  (for  a  more  exten¬ 
sive  description  of  student  modelling  in  ICAI,  see  Ohlsson,  1986). 

Other  methods  for  inferring  cognitive  state  that  are  often  used  in  the 
areas  of  educational  and  cognitive  psychology  include  multidimensional 
scaling  (Kruskal,  1964)  and  hierarchical  clustering  (Johnson,  1967).  These 
methods  are  based  on  an  analysis  of  distance  measures  between  pairs  of 
items.  This  information  can  be  obtained  from  a  variety  of  tasks  such  as 
similarity  judgements,  verification  reaction  times,  number  of  items  inter¬ 
vening  between  two  recalled  items,  etc.  Multidimensional  scaling  places 
items  in  an  N-dimensiona 1  space  in  such  a  way  that  pair-wise  distances  are 
preserved.  Hierarchical  clustering  produces  a  tree  structure  with  nested 
clusters  representing  the  distance  between  items.  The  output  from  these 
analyses  can  then  be  examined  for  the  presence  of  cognitive  structure  that 
is  consistent  with  the  organization  of  concepts  inherent  in  a  particular 
domain.  Although  the  methods  for  inferring  cognitive  state  in  intelligent 
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tutoring  systems  differ  somewhat  from  those  used  in  cognitive  and  educa¬ 
tional  psychology,  they  are  similar  in  that  they  focus  on  qualitative 
aspects  of  knowledge  (e.g.  what  the  knowledge  is,  how  it  is  organized,  and 
how  it  is  used)  rather  than  on  quantitative  aspects  of  knowledge.  More¬ 
over,  they  are  all  guided  by  the  assumption  that  performance  can  be  under¬ 
stood  best  in  the  context  of  the  current  state  of  the  underlying  cognitive 
structures  and  processes  that  mediate  this  performance. 

The  present  research  focuses  on  a  novel  method  of  inferring  cognitive 
structure  called  the  "ordered  tree  technique."  This  technique  was  intro¬ 
duced  by  Reitman  &  Rue  ter,  (1980)  as  a  means  to  recover  consistent  order¬ 
ing  of  to-be- remembered  items  (TBR)  in  a  mutitrial  cued  and  non-cued 
recall  task;  it  has  since  been  extended  to  an  item  organization  task  (c.f. 
Navah-Benjamin ,  McKeachie,  Lin,  &  Tucker,  1986).  A  basic  assumption  un¬ 
derlying  this  technique  is  that  consistent  ordering  of  items  is  a  manifes¬ 
tation  of  the  organization  of  the  cognitive  structures  representing  those 
items  and  of  the  cognitive  processes  operating  on  these  structures.  Spe¬ 
cifically,  items  in  memory  are  organized  into  chunks,  and  all  items  from 
one  chunk  are  produced  before  items  from  another  chunk.  Chunks  are  fur¬ 
ther  organized  into  a  hierarchical  tree  with  higher  level  nodes  in  the 
tree  representing  a  mental  code  for  a  complete  chunk.  For  example,  a 
chunk  consisting  of  the  items  "shoe,"  "glove,"  and  "sweater"  might  be 
represented  by  the  higher  level  node  "articles  of  clothing."  The  ordering 
of  items  during  production  is  a  function  of  this  hierarchical  cognitive 
structure  and  of  the  traversal  rules  for  the  structure.  These  traversal 
rules  may  specify  unidirectional  access  to  items  within  a  chunk  (items  are 
always  produced  in  one  order),  bidirectional  access  (items  are  produced  in 
one  order  or  its  reverse),  or  nondirec tional  access  (items  are  produced  in 
any  order) . 

Using  the  ordered  tree  technique,  it  is  possible  to  recover  experi¬ 
menter-imposed  organizations  for  items  as  well  as  in terpre table  subjective 
organizations.  Further,  chunk  boundaries  in  the  ordered  tree  correspond 
well  to  pauses  in  recall  (Reitman  &  Rueter,  1980).  Recent  studies  have 
shown  that  the  technique  can  differentiate  between  individuals  with  dif¬ 
ferent  skill  levels  in  a  particular  domain  (McKeithen,  Reitman,  Rueter,  & 
Hirtle,  1981)  and  can  show  changes  in  cognitive  organization  for  the  same 
individual  as  learning  occurs  (Navah-Tenjamin,  et  al . ,  1986).  For  exam¬ 
ple,  McKeithen,  et  al.  (1981)  showed  that  novice  programmers'  ordered 
trees  for  reserved  words  in  ALGOL  were  very  different  from  intermediate 
and  expert  programmers'  trees.  Novice  trees  contained  evidence  of 
chunking  by  simple  rules  such  as  orthography  (word  length),  first  letter 
(alphabet),  and  story  construction,  whereas  intermediate  and  expert  trees 
contained  chunks  that  were  relevant  only  in  the  ALGOL  context.  In  addi¬ 
tion,  experts'  trees  were  more  similar  to  each  other  than  were  novices' 
trees  suggesting  that  experts  were  using  similar  cognitive  representations 
to  organize  items.  In  another  study  using  the  ordered  tree  technique, 
Navah-Benjamin,  et  al.  (1986)  showed  that  students'  trees  for  developmen¬ 
tal  psychology  concepts  became  more  organized  and  more  similar  to  the  in¬ 
structor's  tree  over  the  course  of  a  semester.  Moreover,  the  degree  of 
organization  and  similarity  attained  by  the  students  at  the  end  of  the 
course  were  correlated  with  their  final  grades. 
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In  the  present  study,  we  used  the  ordered  tree  technique  to  assess 
changes  in  cognitive  structure  for  a  high  level  computer  command  language 
as  expertise  in  using  that  language  was  aquired.  Computer  command  lan¬ 
guages  consist  of  a  set  of  procedures  that  individuals  can  use  to  perform 
specific  tasks.  These  languages  employ  a  command-execute  cycle  (Moran, 

1981)  that  differs  somewhat  from  the  program-run-execute  cycle  of  tradi¬ 
tional  programming  languages.  Thus,  use  of  these  languages  does  not  re¬ 
quire  programming  skill  (although  many  offer  advanced  features  for  more 
knowledgeable  individuals).  Some  common  examples  of  command  languages  are 
those  used  for  text-editing,  spreadsheet  construction  and  manipulation, 
and  data  base  design  and  manipulation.  In  addition,  command  languages  are 
used  in  all  military  computer  systems  capable  of  performing  a  set  of  tasks 
in  a  manner  that  is  under  the  control  of  an  operator. 

Lotus  1-2-3,  a  menu-based  command  language  for  creating  and  manipulat¬ 
ing  spreadsheets,  was  used  as  the  task  domain.  Lotus  allows  a  user  to 
create  a  spreadsheet  by  inputing  data  into  a  two-dimen tional  grid  space 
displayed  on  a  computer  screen  (see  Figure  1)  and  to  manipulate  this  data 
using  a  variety  of  commands  and  functions  (see  Figure  2).  One  goal  of 
this  research  was  to  gain  some  understanding  of  how  skill  in  using  this 
language  is  acquired  and  what  software  design  features  facilitate  or  hin¬ 
der  this  process.  This  information  will  be  important  in  developing  better 
training  for  command  language  use  in  military  settings.  A  second  goal, 
and  the  one  that  is  the  focus  of  this  paper,  was  to  determine  whether  the 
ordered  tree  technique  can  illuminate  changes  in  cognitive  structure  dur¬ 
ing  the  acquisition  of  procedural  computer  skills. 

METHOD 

Twenty-four  MBA  students  at  the  University  of  Michigan  who  had  no 
previous  experience  with  Lotus  1-2-3  received  a  1  1/2  hour  tutorial  on 
basic  use  of  Lotus  and  were  subsequently  required  to  use  the  software  in  a 
class  project.  The  tutorial  consisted  of  a  lecture  and  a  spreadsheet 
construction  practical  exercise  using  a  Burroughs  personal  computer  sys¬ 
tem.  The  class  project  consisted  of  construction  of  a  relatively  complex 
financial  spreadsheet.  The  students  were  allowed  to  work  in  groups  for 
this  project.  One  week  after  the  tutorial  and  at  the  end  of  the  semester 
following  the  completion  of  the  class  project,  each  student  participated 
in  a  2  hour  data  collection  session  in  which  their  organization  of  Lotus 
knowledge  was  assessed.  Data  was  collected  at  the  same  time  for  an  expert 
who  was  using  the  Lotus  1-2-3  macro  facility  to  develop  decision  aids. 

The  ordered  tree  technique  was  used  to  determine  the  organization  of 
Lotus  knowledge.  Subjects  were  given  a  set  of  cards  containing  32  words 
selected  to  represent  a  subset  of  procedural  steps  and  related  concepts  in 
Lotus.  They  were  asked  to  use  their  knowledge  of  Lotus  to  organize  the 
words  so  that  terms  that  were  closely  related  in  Lotus  were  close  together 
in  their  organization.  They  listed  this  organization  sequentially  in  a 
test  booklet,  shuffled  the  cards,  and  repeated  the  procedure  9  additional 
times.  The  test  booklet  contained  10  pages  with  32  blank  lines  for  the  32 
Lotus  terms.  Eight  of  the  10  pages  began  with  a  different  word  from  the 
subset  of  Lotus  terms,  and  two  pages  began  with  an  empty  line.  Subjects  were 
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told  that  if  a  page  began  with  a  prespecified  item,  they  were  to  begin 
listing  their  organization  with  words  that  were  related  to  this  word;  if 
the  first  line  was  blank  they  were  told  that  they  could  begin  with  any 
word  they  chose.  This  procedure  yielded  10  orderings  of  the  Lotus  terms, 
which  were  then  submitted  to  the  ordered  tree  algorithm  to  determine  con¬ 
sistent  orderings  of  words  within  the  data.  This  algorithm  also  produced 
a  measure,  PRO,  of  the  degree  of  organization  within  a  given  tree  struc¬ 
ture.  The  lower  this  measure,  the  higher  the  organization  in  a  tree. 

Ability  to  use  the  Lotus  software  was  assessed  using  two  tasks.  In  a 
spreadsheet  construction  task,  subjects  received  a  spreadsheet  printout 
and  specifications  and  were  asked  to  construct  the  spreadsheet  themselves 
as  accurately  as  possible.  Following  the  construction  task,  they  were 
asked  to  modify  the  spreadsheet  they  had  constructed  using  several  spe¬ 
cific  procedures.  Different  spreadsheets  were  used  for  each  of  the  two 
sessions.  However,  these  spreadsheets  were  constructed  using  the  same 
template  so  that  they  were  visually  similar  and  required  the  same  proce¬ 
dures  to  complete  (see  Figure  3).  During  performance  of  the  spreadsheet 
construction  and  modification  tasks,  keystroke  sequences  and  times  were 
recorded  yielding  a  sequentially  ordered  protocol  for  the  procedures 
attempted  and  successfully  or  unsuccessfully  completed.  Four  procedures 
were  selected  for  analysis:  widening  column  widths,  inserting  rows  and 
columns,  and  constructing  formulas  and  functions.  The  first  two  proce¬ 
dures  required  use  of  the  LOTUS  menu,  the  latter  two  did  not. 

RESULTS 

Ordered  trees  for  the  expert  and  two  typical  novices  are  presented  in 
Figures  4-6.  Chunk  boundaries  or  nodes  are  indicated  by  small  circles,  and 
chunk  directionality  is  indicated  by  the  arrow  over  the  branches  of  a 
node.  The  absence  of  an  arrow  indicates  a  non-directional  chunk. 

It  is  clear  from  Figure  4  that  there  is  a  great  amount  of  organization 
in  the  expert's  ordered  trees  for  Sessions  1  and  2  (PRO's  =  24  and  25, 
respectively,  out  of  a  maximum  of  117).  Further,  although  the  trees  are  not 
identical,  the  chunks  they  contain  overlap  considerably.  The  degree  of 
overlap  or  similarity  can  be  determined  by  computing  the  number  of 
non-trivial  chunks  the  two  trees  have  in  common  (McKeithen,  et  al.,  1981). 
This  is  given  by  the  equation: 

ln(intersectlon  of  trees  +  l)/In(unlon  of  trees  +  1) 

This  equation  produces  a  similarity  value  of  .73  for  the  two  expert  trees 
out  of  a  maximum  score  of  1.00.  Thus,  the  expert  seems  to  have  used  a  simi¬ 
lar  cognitive  representation  to  organize  the  Lotus  terms  in  Sessions  1  and 
2. 

The  expert's  trees  for  both  sessions  reveal  organization  that  is  highly 
meaningful  in  the  Lotus  context.  More  importantly,  the  chunks  represent  both 
related  concepts  and  frequent  procedural  sequences  in  Lotus.  For  example, 
the  chunks  LABEL  -  PREFIX  -  "  -  and  MENU  -  VALUE  -  POINT  represent  con¬ 
cepts  that  are  associated  in  Lotus  but  have  no  relationship  outside  of 
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this  command  language.  The  chunks  DELETE  -  INSERT  -  ROW  -  COLUMN  -  WIDTH 
and  COPY  -  MOVE  -  FROM  -  TO  represent  procedural  sequences.  Interestingly, 
the  normative  directional  association  between  TO  -  FROM  has  been  reordered 
to  correspond  to  the  correct  Lotus  procedural  sequence  FROM  -  TO.  Finally, 
in  the  Session  2  ordered  tree,  a  high  level  unidirectional  chunk  repre¬ 
senting  the  Lotus  menu  structure  appears.  The  directionality  of  this 
chunk  is  congruent  with  what  would  be  observed  if  one  moved  sequentially 
through  each  top  level  menu  commmand  and  its  sub-menu  commands  (See  Fig¬ 
ure  2  for  a  graphic  representation  of  the  LOTUS  menu  structure). 

The  ordered  trees  in  Figure  5  were  produced  by  a  novice  whose  similar¬ 
ity  to  the  expert  increased  substantially  over  the  two  sessions  (.00  to 
.59).  In  the  first  session,  the  novice  tree  is  poorly  organized  (PRO  = 

109)  and  bears  no  similarity  to  the  expert  tree.  In  session  2,  however, 
greater  organization  has  emerged  (PRO  =42)  and  this  organization  begins 
to  resemble  that  of  the  expert.  For  example,  the  terms  INSERT  -  DELETE 
have  been  added  to  the  original  chunk  and  the  new  chunk  begins  to  resemble 
the  expert  chunk.  In  addition,  the  novice  has  apparently  gained  a 
rudimentary  representation  of  the  menu  structure  as  shown  by  the 
chunk  /  -  MENU  -  WORKSHEET  -  RANGE  -  MOVE  -  COPY.  Unlike  the  expert's 
representation  of  the  menu,  however,  the  novice's  representation  contains 
only  top  level  menu  commands.  The  Session  2  tree  also  shows  that  the 
novice  has  some  misconceptions  about  Lotus.  The  symbol  @  is  used  to  indi¬ 
cate  a  FUNCTION  in  Lotus,  but  the  novice  has  placed  it  with  LABEL  -  PRE¬ 
FIX.  LABEL  -  PREFIX  should  only  be  associated  with  the  symbols  V  and  ". 
Thus,  this  novice  apparently  does  not  have  a  clear  representation  for  the 
LABEL  -  PREFIX  concept.  In  general,  this  novice's  emerging  representation 
can  be  characterized  as  relatively  accurate  but  incomplete  at  this  stage 
in  learning.  The  ordered  trees  for  the  second  novice  (see  Figure  6)  also 
demonstrate  an  increase  in  organization  (PRO  =  117  to  109)  and  similarity 
to  the  expert  (.00  to  .31),  but  these  increases  are  minimal.  Further,  the 
chunks  present  in  this  novice's  Session  2  tree  could  be  constructed  using 
prior  associations  as  well  as  Lotus  knowledge. 

These  tree  structures  for  an  expert  and  two  typical  novices  with  dif¬ 
fering  levels  of  similarity  to  the  expert  show  that  the  ordered  tree  tech¬ 
nique  can  recover  differences  in  cognitive  representations  for  Individuals 
with  different  skill  levels  and  for  the  same  individual  after  some  learn¬ 
ing  of  the  skill  has  occured.  In  general,  the  expert  showed  greater  or¬ 
ganization  than  the  group  of  novices  in  both  Session  1  (PRO  -  24  vs  70) 
and  Session  2  (PRO  -  25  vs  54  )  although  organization  did  increase  for  the 
novices  between  the  two  sessions  (PRO  -  70  vs  54).  To  determine  whether 
this  increase  in  organization  reflected  relationships  that  were  relevant 
to  Lotus,  the  novice  similarity  scores  for  Sessions  1  and  2  were  analyzed. 
These  means  were  .26  and  .36,  respectively,  and  this  increase  in  similar¬ 
ity  was  significant,  2](1»23)  =  6.92,  MSe  =  .017,  ^  <  .01.  Thus,  the 
novices'  organization  of  Lotus  terms  increased  over  sessions  and  this 
organization  also  became  more  like  that  of  an  expert.  This  suggests  that 
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Figure  4.  Expert  ordered  trees  for  sessions 


Figure  5.  Novice  ordered  trees  tor  session 
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FiRure  6.  Novice  ordered  trees  for  session 
(Small  increase  in  similarity  to 


the  novices  were  beginning  to  develop  an  underlying  cognitive  representa¬ 
tion  for  these  items  that  reflected  procedural  sequences  and  related  con¬ 
cepts  in  Lotus. 

Two  additional  analyses  were  conducted  to  determine  whether  there  was  a 
relationship  between  the  ordered  trees  obtained  for  novices  and  their 
ability  to  use  the  Lotus  software.  Error  rates  for  the  four  procedures 
mentioned  previously  were  used  as  the  measure  of  ability.  For  the  first 
analysis,  the  subjects  were  divided  into  subgroups  on  the  basis  of  whether 
their  similarity  scores  were  higher  or  lower  than  the  overall  group  median 
for  Sessions  1  and  2.  Subjects  in  the  LOW  group  had  similarity  scores 
lower  than  the  median  in  both  sessions  (N=7);  subjects  in  the  IMPROVE  group 
had  scores  lower  than  the  median  in  Session  1,  but  higher  than  the  median 
in  Session  2  (N=5);  subjects  in  the  REGRESS  group  had  scores  higher  in  the 
first  session,  but  lower  in  the  second  session  (N=6);  and  subjects  in  the 
HIGH  group  had  similarity  scores  higher  than  the  median  in  both  sessions 
(N=6).  The  mean  number  of  errors  for  these  groups  for  the  four  procedures 
are  presented  in  Table  1.  A  4  (Group)  X  2  (Session)  X  4  (Procedure)  ANOVA 
for  this  data  revealed  main  effects  of  group,  F(.2,20)  =  6.91,  MSe  =  .133, 

£  <  .022,  and  procedure,  £(3,60)  =  10.32,  MSe  =.137,  ^  <  .001.  No  other 
main  or  interaction  effects  were  significant.  These  results  show  that  the 
HIGH  similarity  group  made  substantially  fewer  errors  than  the  LOW,  IM¬ 
PROVE,  and  REGRESS  groups,  but  that  there  was  little  difference  between  these 
latter  three  groups.  Further,  the  overall  increase  in  similarity  between 
Sessions  1  and  2  does  not  appear  to  be  matched  by  a  corresponding  decrease 
in  the  number  of  errors  made,  nor  is  there  an  interaction  that  would  sug¬ 
gest  that  errors  decreased  for  some  groups  but  not  for  others.  Thus, 
although  the  ordered  trees  show  some  correspondence  to  performance,  this 
correspondance  is  not  perfect. 

Table  1 

Mean  proportion  of  errors  by  group,  session,  and  task. 


Session 

Group 

Procedure 

Low 

Improve 

Regress 

High 

Mean 

1 

COL  WIDTH 

.30 

.21 

.17 

.07 

.19 

INSERT 

.51 

.36 

.25 

.14 

.32 

FORMULA 

.56 

.41 

.59 

.33 

.48 

FUNCTION 

.53 

.80 

.68 

.33 

.57 

Mean 

.47 

.44 

.42 

.22 

2 

COL  WIDTH 

.38 

.07 

.32 

.09 

.23 

INSERT 

.32 

.50 

.32 

.03 

.29 

FORMULA 

.79 

.52 

.40 

.17 

.48 

FUNCTION 

.58 

.63 

.80 

.36 

.59 

Mean 

.52 

.43 

.46 

.16 
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To  obtain  additional  information  about  the  relationship  between  the 
ordered  trees  and  performance,  correlations  between  similarity  and  error 
rates  were  obtained.  These  correlations  are  presented  in  Table  2.  As  the 
table  shows,  the  only  significant  correlation  between  similarity  and  error 
rate  in  Session  1  was  for  the  INSERT  procedure.  For  Session  2,  however, 
there  were  significant  correlations  between  these  factors  for  the  proce¬ 
dures,  widen  column  width,  construct  formula,  and  construct  function.  This 
suggests  that  after  some  skill  has  been  acquired,  similarity  to  the  expert 
may  be  related  to  ability  to  use  the  software.  This  finding  parallels 
previous  findings  that  similarity  is  correlated  with  final  course  grades 
but  not  with  grades  in  the  beginning  of  a  course  (Navah-Benjamin,  et  al., 
1986) . 

These  results  are  consistent  with  earlier  studies  (McKeithen,  et  al., 
1981;  Navah-Benjamin,  1986)  in  showing  that  the  ordered  tree  technique  can 
reveal  differences  in  cognitive  structure  for  individuals  with  differing 
procedural  skill  levels  and  for  the  same  individual  as  skill  increases. 
There  appears  to  be  a  relationship  between  the  cognitive  representation 
described  by  these  ordered  trees  and  procedural  ability,  especially  for 
Session  2.  However,  this  relationship  is  by  no  means  as  clear  as  desired. 
There  are  a  number  of  possible  reasons  for  this  finding.  Perhaps  the  Lotus 
terms  chosen  were  not  representative  of  the  knowledge  used  for  the  proce¬ 
dures  analyzed  (i.e.,  performance  was  mediated  by  knowledge  structures 
other  than  those  represented  in  the  ordered  trees)  or  perhaps  the  measures 
of  skill  and  ability  to  use  Lotus  were  inflated  because  subjects  could  use 
contextual  cues  present  in  the  software  interface  (e.g.  the  menu  struc¬ 
ture)  to  guide  performance.  These  cues  were  not  present  during  perform¬ 
ance  of  the  organization  task.  Alternately,  it  may  be  the  case  that  a 
well-formed  cognitive  representation  is  not  necessary  for  some  minimal 
ability  to  use  the  software  to  be  demonstrated. 

Table  2 

Correlations  between  similarity  and  error  rate  for  each  procedure. 


Session  Procedure  Correlation 


COL  WIDTH 

-.11 

INSERT 

-.46 

** 

FORMULA 

-.07 

FUNCTION 

-.05 

COL  WIDTH 

-.47 

★  * 

INSERT 

-.02 

FORMULA 

-.39 

* 

FUNCTION 

-.40 

★ 
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Despite  the  absence  of  a  strong  relationship  between  performance  and 
the  tree  structures  obtained  in  this  experiment,  the  ordered  tree  tech¬ 
nique  appears  to  produce  a  valid  representation  of  the  underlying  cogni¬ 
tive  representations  for  Lotus.  This  technique  has  also  been  proven 
effective  for  inferring  cognitive  representations  in  a  number  of  other 
domains,  including  programming  in  ALGOL  (McKeithen,  et  al.,  1981)  and  the 
psychology  of  aging  (Navah-Benjamin ,  et  al.,  1986).  Future  applications 
for  this  technique  may  be  its  incorporation  into  an  ICAI  system  to  aid 
student  modeling  and  its  use  as  a  "first-cut"  method  to  elicit  and  organ¬ 
ize  expert  knowledge  during  knowledge  acquisition  for  expert  systems. 
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